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Why  AF  SEAM 
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Problem: 


-  AF  programs  late,  over  cost,  &  do  not  provide  the 
performance  expected 

-SECAF  directed  action  to  revitaiize  SE  across  the  AF 

-  No  standard  tooi/method  for  assessing  SE  processes 


Goals: 

-  Promote  consistent  understanding  of  SE 

-  Ensure  core  SE  processes  are  in  piace  and  being 
practiced 

-  Facilitate  sharing  “Best  Practices” 

-  Provide  “Brain  Drain”  insurance 

-  improve  AF  leadership  visibility  into  SE  process  maturity 


Improved  program  performance  &  reduced  technical  risk 


Background 
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•  Original  task:  AFMC  EC  Action  Item 

•  Objective:  “Develop  standard  AF  assessment  model” 

•  Tools  were  in  place  @  4  Centers 

•  12  On-Site  Team  Engagements 

•  Representatives  from  EN  Home  Offices 

•  4  Product  Centers,  3  ALCs,  AEDC,  HQ  AFMC/EN,  CSE 

•  Met  9  times  at  5  different  locations  in  one  year 

•  Conducted  3  baseline  assessments  at  3  Centers 

•  12  Briefings  to  Senior  Leaders 

•  AFMC  Engineering  Council  Meetings  (4) 

•  ALC  EN  Meeting 

•  SAF/AQR(2) 

•  AF  Tech  Leaders  Round  Table 

•  OSD  (AT&L)  &  Boeing  SE  Advisory  Group 

•  National  Research  Council  (National  Academies) 


•  Final  to  AFMC/EN  -  5  Aug  08,  &  Final  to  SAFF/AQR  -  11  Aug  08 


Development  Schedule 
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ID  Process  Areas  (10) 


Define  Process  Area  Goals  (34) 


Defined  Practices  aligned  to  goals  (120) 


Formative  Material  Includes: 

-  Detail  Practice  Description 

-  Practice  Assessment  Criteria 

-  Reference  Materials 


1 - T 


Develop  Assessment  &  Scoring  Methodology 


Develop  &  Implement  Training  Plan 


ID  Baseline  Test  Candidates 


Perform  Baseline  Training  &  Assessments 


Incorporate  Baseline  Test  Feedback  & 
Revise  AF  SEAM 


1 .  Ground  Theater  Air  Control  Systems 

2.  TSAT  at  SMC 

3.  Mission  Planning  Systems 


m 


Coordinate  Final  Version  (Spiral  1) 


DELIVERED  ON  TIME  ! 


Development  Process 


•  Environmental  Scan  Up  Front 

-  External  Benchmarking 

-  Existing  Best  Practices 

•  Collaborative  Reviews/Inputs 

-  Software  Engineering  Institute  (CMMI) 

-  NDIA 

-  AF  HSIO 

-  LHA  Development  Team 

-  TD  1-12 

-  INCOSE 

-  Industry  Partners 


Collaborative  build  -  Included  greater  SE  community 


Model  Construct 
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AF  SEAM 


AF 


Center 

Specific 


Size/ 

Complexity 


Point  in 
Life-Cycle 


>  Center 


J 


Individually  Tailored  SEAM 


^Program 


Tool  Suite 


•  Management  Guide 

•  Assessment  Tool  (Spreadsheet) 

•  Training 

-  Orientation/Overview 

-  Self-Assessment 

-  Validation  Team 


S/  Specific  Practices  Summary 


PA  LEGEND 


90-100% 


65-89% 


SP  LEGEND 


Not  Applicable 


P0rc0ntag0 
( of  r/io-se 

scoref/J 


Spreadsheet  tool  provides  this  output 


Scoring  Roll-Up 


Specific  Practice  Assessment  Resuits 

XXX  Center 


CM  DA  D  M  PP  R  RM  S  TMC  V 


Process  Area 


^  Spiral  2  Considerations 


•  Capability  Enhancement 

-  Re-look  process  areas  for  improvements 

-  Further  refine  assessment  methodology 

-  Strengthen  inclusion  of  software 

-  Capture  and  promulgate  best  practices/lessons  learned 

-  Review  scoring 

-  Examine  potential  use  for  SE  health  assessment 

-  Migrate  to  web-based  piatform 

•  Charter 

-  Estabiish  vision  &  mission 

-  Estabiish  governance 

-  Support  team  by  providing  resources 

-  Signed  @  appropriate  level 

•  Funding 

-  Spiral  2  &  Sustainment 

•  Lead  POC/Steering  Group 

-  Staff  support 

-  Community  of  Interest 

-  Configuration  control 
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Implementation  By  Center 


CENTER 

5  AUG  08 -FEEDBACK 

AAC 

"AAC  began  integrating  AF  SEAM  in  our  established  program  assessment 
process  in  January  2008  and  expects  to  complete  this  integration  in  FY09." 

AEDC 

"We  will  begin  implementing  AF  SEAM  in  October." 

ASC 

"We  are  creating  a  plan  to  migrate  from  our  current  tool  to  SEAM,  tailored 
with  AFMC  and  ASC  specific  areas  of  interest." 

ESC 

"We  have  initiated  tailoring  efforts  to  implement  AF  SEAM  by  the  end  of  the 
calendar  year.  We  will  be  working  closely  with  SMC,  our  acquisition  partner, 
on  the  tailoring  and  implementation  effort." 

OC-ALC 

"Strongly  support,  have  plans  in  place,  ready  to  go!" 

OO-ALC 

"We  are  implementing  now." 

SMC 

"SMC  plans  to  adopt  AF  SEAM  and  comply  with  related  policies." 

WR-ALC 

"We'll  begin  implementation  at  Robins  with  pilot  assessments  in  F-15  and 
Avionics." 

Development  process  yielded  100%  buy-in 


QUESTIONS  ? 


Back  Up  Slides 
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Agenda 
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•  Background 

•  Development  process 

•  AF  SEAM  tool  suite: 

•  Management  guide 

•  Assessment  tool  (Spreadsheet) 

•  Training 

•  Results  reporting 

•  IPT  overarching  concerns 

•  Spiral  2  considerations 


Defining  the  Methodoiogy 


Assessment  Continuum 


•  Hands  Off 

•  Promuigate 
Poiicy 

-  Directives 

-  Instructions 

-  Checklists 

-  Guidance 

•  Expect 
Compliance 


AF  SEAM 

-  Collaborative 
&  inciusive 

-  Leanest  possible 

best  practices  “Must  Dos” 

-  Ciearly  stated  expectations 

-  Program  team  &  assessor 
team 

-  Training 

Self-assessment 
of  program  with 
optional  validation 


Hands  On 

Comprehensive 

Continuous 

Process 

improvement 

-  Highly  detailed 
process  books 

-  Training 

independent 

Assessment 

-  Deep  dives 


Assessment  methods  that  balance  time  &  effectiveness 


^  Generic  Practices  Summary 

- m 


Spreadsheet  tool  provides  this  output 


Scoring  Roll-Up 
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Generic  Practice  Assessment  Resuits 

XXX  Center 


Practice  Area 
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IPT  Overarching  Concerns 


Policy/Guidance 

-  Integrate  into  AF  Instructions,  UCI  checklists,  policy  directives,  & 
SEP  guidance 

-  Required  to: 


•  Transition  from  current  implementation  state  to  standardized 
execution  AF-wide 

•  Achieve  AF-level  standardization  of  SE  processes 

•  Ensure  that  near-term  demands  do  not  override  long-term  benefits 


-  Develop  timetable  for  roll-out 

-  Need  to  preserve  AF  SEAM  as  a  process  improvement  tool 

•  Culture  change  required 

•  Accurate  results  rely  upon  candid  program/project  team  responses 
(non-attribution) 

•  Tiered  results  reporting  delivers  accurate  &  actionable  information  to 
each  leadership  level 


•  Interaction  of  AF  SEAM  w/  other  assessment  methodologies 
&  tools 

-  POPS,  SMART,  CMMI,  LHA,  etc. 


Why  AF  SEAM 
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Problem: 


-  AF  programs  late,  over  cost,  &  do  not  provide  the 
performance  expected 

-SECAF  directed  action  to  revitaiize  SE  across  the  AF 

-  No  standard  tooi/method  for  assessing  SE  processes 


Goals: 

^  Promote  consistent  understanding  of  SE 

Q  Ensure  core  SE  processes  are  in  place  and  being 
oracticed 

B  Facilitate  sharing  “Best  Practices” 

B  Provide  “Brain  Drain”  insurance 

B  Improve  AF  leadership  visibility  into  SE  process  maturity 


Improved  program  performance  &  reduced  technical  risk 


Team  Members 


Center 

Members 

AAC 

Ian  Talbot 

AEDC 

Neil  Peery,  Maj  Mark  Jenks 

ASC 

Gary  Bailey 

AF  CSE 

Randy  Bullard,  Rich  Freeman 

HQAFMC 

Caroline  Buckey 

ESC 

Bob  Swarz,  Bruce  Allgood 

OC-ALC 

Cal  Underwood,  Bill  Raphael 

OO-ALC 

Jim  Belford,  Mahnaz  Maung 

SMC 

Linda  Taylor 

WR-ALC 

Jim  Jeter,  Ronnie  Rogers 

Process  Areas 


# 

Symbol 

Process 

1 

CM 

Configuration  Management 

2 

DA 

Decision  Analysis 

3 

D 

Design 

4 

M 

Manufacturing 

5 

PP 

Project  Planning 

6 

R 

Requirements 

7 

RM 

Risk  Management 

8 

S 

Sustainment 

9 

TMC 

Technical  Management  &  Control 

10 

V 

Verification  &  Validation 

Generic  Practices 


# 

Practice  Description 

GP1 

Description  of  process 

GP2 

Plans  for  performing  the  process 

GPS 

Adequate  resources  for  performing  the  process 

GP4 

Responsibility  &  authority  for  performing  the  process 

GPS 

Train  the  people  performing  the  process 

GP6 

Monitor  &  control  the  process 

GP7 

Review  activities,  status,  &  results  of  the  process 

Proof  of  Concept 
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•  Base  ESC  pilot  on  ENweb  &  open  source  survey  engine 

•  Make  tool/code  available  to  all  Centers 

•  Will  include: 

-  Basic  implementation  of  aii  rule  sets  and  standards 

-  Rudimentary  access  control  of  assessment  results 

-  All  presentation  formats  deveioped  for  AF  SEAM,  downloadabie 
for  incorporation  into  briefings 

-  Data  entry  via  a  web  interface 

-  Uploading  of  supporting  artifacts 

•  Form  AF-wide  Working  Group  to  discuss  larger 
implementation 

•  Initiate  dialog  with  554  ELSW  to  address/develop  courses  of 
action  for  an  acquisition  program  to  develop  an  AF  SEAM 
toolset 


Survey  Format  Concept 


Requirements  (R) 

The  purpose  of  the  Requirements  process  area  is  to  develop  and  analyze  operational  user^  product,  and  product-component  requirements, 
to  assure  consistency  between  those  require-ments  and  the  project's  technical  plans  and  work  products  and  to  manage  requirements 

evolu-tion  through  the  life  cycle  of  the  product. 

"^Have  comprehensive  operational  use  cases  and  key  test  scenarios  been  developed  and  coordinated  with 
the  end  user,  acquirer  and  the  operational  test  agency? 

O  Yes 

Ono 

"^Have  comprehensive  operational  use  cases  and  key  test  scenarios  been  developed  and  coordinated  with 
the  end  user,  acquirer  and  the  operational  test  agency? 

O  Yes 
O  No 

*Are  requirements  clearly  and  succinctly  stated? 

O  Yes 
Ono 

*  Is  there  a  viable  closure  plan  for  determining  when  all  technical  requirements  have  been  defined  and 

analyzed? 

O  Yes 
Ono 

"^What  process  is  used  to  ensure  that  validated  threat  information  and  operational  concepts  are 

incorporated  into  requirements  analysis? 

_ (upload  document) _ 


Presentation  Concept 


Assessments 


[ed  i  t] 


Watchlists  (all  CMD  level}^ 

■  350th  a  I  55 1st  a  I  653rd  ^  |  554th  ^  |  CPSG® 

STP:  All  Wings  ^ 

■  350thg  I  551sta  I  653rdl  |  554thl  |  Other© 


■  CMMI  All  Wings© 

■  350th©  I  850th©  |  950th@ 

■  55l3t©  I  651st©  I  751st©  |  851st©  |  951st© 

■  554st©  I  642nd©  |  653rd©  |  754th© 

■  639th©  I  640th©  |  KC©  |  Kl©  [none)  |  KN©  |  KZ© 

■  753rd©  I  853rd©  |  CP3G©  |  Intel©  (none)  |  Other© 
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Rolf-up 

Requirements  Config  Mgmt  Sustainment  Risk  Mgmt 
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AF  SEAM 


AF  SEAM  Processes 

•  Requirements 

•  Design 

•  V&V 

•  Decision  Analysis 

•  Configuration  Mgmt 

•  Risk  Mgmt 

•  Project  Planning 

•  Sustainment 

•  Manufacturing 

•  Tech  Mgmt  &  Ctrl 

•  Generic  Practices 


CMMI  Color  Legend:  Green  =  Covered,  Yellow  = 
Partially,  Red  =  Not  Covered 


CMMI-ACQv12 


CMMI-ACQ  Processes  v 


REQM  -  Requirements  Management  (RM)  2 

MA  -  Measurements  &  Analysis 
PMC  -  Project  Monitoring  &  Control 
PP  -  Project  Planning 

PPQA-  Process  and  Product  Quality  Assurance 
SSAD  -  Solicitation  &  Supplier  Agreement  Dev 
CM  -  Configuration  Management 

DAR  -  Decision  Analysis  and  Resolution  3 

AM  -  Agreement  Management 

ARD  -  Acq  Requirements  Development 

ATM  -  Acq  Technical  Management 

VAL- Acq  Validation 

VER  -  Acq  Verification 

OPD  -  Organizational  Process  Definition 

OPF  -  Organizational  Process  Focus 

IPM  -  Integrated  Project  Management  (IPPD) 

RSKM  -  Risk  Management 

OT  -  Organizational  Training 

OPP  -  Organizational  Process  Performance 

QPM  -  Quantitative  Project  Management  ^ 

OID  -  Organizational  Innovation  &  Deployment 

CAR  -  Causal  Analysis  &  Resolution  ^ 


CMMI  Maturity  Levels: 

1  Initial,  2  Managed,  3  Defined,  4  Quantitatively  Managed,  5  Optimizing 


V  AF  seam  - 


AF  SEAM  Processes 

•  Requirements 

•  Design 

•  V&V 

•  Decision  Analysis 

•  Configuration  Mgmt 

•  Risk  Mgmt 

•  Project  Planning 

•  Sustainment 

•  Manufacturing 

•  Tech  Mgmt  &  Ctrl 

•  Generic  Practices 

CMMI  Color  Legend:  Green  =  Covered,  Yellow  = 

Partially,  Red  =  Not  Covered 

CMMI  Maturity  Levels: 

1  Initial,  2  Managed,  3  Defined,  4  Quantitatively  Managed,  5  Optimizing 


CMMI-DEVvi.2 


Process  Area 

Causal  Analysis  and  Resolution  5 

Configuration  Management  2 

Decision  Analysis  and  Resolution  3 

Integrated  Project  Management  +IPPD  3 

Measurement  and  Analysis  2 

Organizational  Innovation  and  Deployment  5 

Organizational  Process  Definition  +IPPD  3 

Organizational  Process  Focus  3 

Organizational  Process  Performance  4 

Organizational  Training  3 

Product  Integration  3 

Project  Monitoring  and  Control  2 

Project  Planning  2 

Process  and  Product  Quality  Assurance  2 

Quantitative  Project  Management  4 

Requirements  Development  3 

Requirements  Management  2 

Risk  Management  3 

Supplier  Agreement  Management  2 

Technical  Solution  3 

Validation  3 


Verification 


3 


